<!DOCTYPE html><html lang="zh-CN" data-theme="light"><head><meta charset="UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Hadoop3.X分布式运行环境搭建手记 | 全栈进阶那些事</title><meta name="keywords" content="Hadoop"><meta name="author" content="liuyuantao"><meta name="copyright" content="liuyuantao"><meta name="format-detection" content="telephone=no"><meta name="theme-color" content="#ffffff"><meta name="description" content="1.准备虚拟机（关闭防火墙、静态ip、主机名称）1.1首先使用VMware安装CentOS7.4虚拟机安装好虚拟机后设置好静态IP(192.168.126.120,设置DNS)，hostname，执行yum update更新软件   这里由于进行了规划，所以提前设置了hosts，防止ping hostname的时候不同  1.2安装软件安装vim，在&#x2F;usr&#x2F;lib&#x2F;jvm目录安装好jdk1.8.">
<meta property="og:type" content="article">
<meta property="og:title" content="Hadoop3.X分布式运行环境搭建手记">
<meta property="og:url" content="https://liuyuantao.gitee.io/2020/11/07/fd53f751c2eb.html">
<meta property="og:site_name" content="全栈进阶那些事">
<meta property="og:description" content="1.准备虚拟机（关闭防火墙、静态ip、主机名称）1.1首先使用VMware安装CentOS7.4虚拟机安装好虚拟机后设置好静态IP(192.168.126.120,设置DNS)，hostname，执行yum update更新软件   这里由于进行了规划，所以提前设置了hosts，防止ping hostname的时候不同  1.2安装软件安装vim，在&#x2F;usr&#x2F;lib&#x2F;jvm目录安装好jdk1.8.">
<meta property="og:locale" content="zh_CN">
<meta property="og:image" content="https://oscimg.oschina.net/oscnet/c646ab2c9196c3eacc49b81b10d47f7a191.jpg">
<meta property="article:published_time" content="2020-11-07T05:33:37.000Z">
<meta property="article:modified_time" content="2021-11-12T07:45:20.873Z">
<meta property="article:author" content="liuyuantao">
<meta property="article:tag" content="Hadoop">
<meta name="twitter:card" content="summary">
<meta name="twitter:image" content="https://oscimg.oschina.net/oscnet/c646ab2c9196c3eacc49b81b10d47f7a191.jpg"><link rel="shortcut icon" href="/img/favicon.png"><link rel="canonical" href="https://liuyuantao.gitee.io/2020/11/07/fd53f751c2eb"><link rel="preconnect" href="//cdn.jsdelivr.net"/><link rel="preconnect" href="//hm.baidu.com"/><link rel="preconnect" href="//busuanzi.ibruce.info"/><link rel="stylesheet" href="/css/index.css"><link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@fortawesome/fontawesome-free/css/all.min.css" media="print" onload="this.media='all'"><link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/node-snackbar/dist/snackbar.min.css" media="print" onload="this.media='all'"><script>var _hmt = _hmt || [];
(function() {
  var hm = document.createElement("script");
  hm.src = "https://hm.baidu.com/hm.js?d43383137144f76a37f1074576bc674e";
  var s = document.getElementsByTagName("script")[0]; 
  s.parentNode.insertBefore(hm, s);
})();
</script><script>const GLOBAL_CONFIG = { 
  root: '/',
  algolia: undefined,
  localSearch: {"path":"search.xml","languages":{"hits_empty":"找不到您查询的内容：${query}"}},
  translate: {"defaultEncoding":1,"translateDelay":0,"msgToTraditionalChinese":"繁","msgToSimplifiedChinese":"简"},
  noticeOutdate: undefined,
  highlight: {"plugin":"highlighjs","highlightCopy":true,"highlightLang":true,"highlightHeightLimit":200},
  copy: {
    success: '复制成功',
    error: '复制错误',
    noSupport: '浏览器不支持'
  },
  relativeDate: {
    homepage: true,
    post: true
  },
  runtime: '天',
  date_suffix: {
    just: '刚刚',
    min: '分钟前',
    hour: '小时前',
    day: '天前',
    month: '个月前'
  },
  copyright: {"limitCount":500,"languages":{"author":"作者: liuyuantao","link":"链接: ","source":"来源: 全栈进阶那些事","info":"著作权归作者所有。商业转载请联系作者获得授权，非商业转载请注明出处。"}},
  lightbox: 'mediumZoom',
  Snackbar: {"chs_to_cht":"你已切换为繁体","cht_to_chs":"你已切换为简体","day_to_night":"你已切换为深色模式","night_to_day":"你已切换为浅色模式","bgLight":"#49b1f5","bgDark":"#2d3035","position":"top-right"},
  source: {
    jQuery: 'https://cdn.jsdelivr.net/npm/jquery@latest/dist/jquery.min.js',
    justifiedGallery: {
      js: 'https://cdn.jsdelivr.net/npm/justifiedGallery/dist/js/jquery.justifiedGallery.min.js',
      css: 'https://cdn.jsdelivr.net/npm/justifiedGallery/dist/css/justifiedGallery.min.css'
    },
    fancybox: {
      js: 'https://cdn.jsdelivr.net/npm/@fancyapps/fancybox@latest/dist/jquery.fancybox.min.js',
      css: 'https://cdn.jsdelivr.net/npm/@fancyapps/fancybox@latest/dist/jquery.fancybox.min.css'
    }
  },
  isPhotoFigcaption: true,
  islazyload: false,
  isanchor: true
}</script><script id="config-diff">var GLOBAL_CONFIG_SITE = {
  title: 'Hadoop3.X分布式运行环境搭建手记',
  isPost: true,
  isHome: false,
  isHighlightShrink: false,
  isToc: true,
  postUpdate: '2021-11-12 15:45:20'
}</script><noscript><style type="text/css">
  #nav {
    opacity: 1
  }
  .justified-gallery img {
    opacity: 1
  }

  #recent-posts time,
  #post-meta time {
    display: inline !important
  }
</style></noscript><script>(win=>{
    win.saveToLocal = {
      set: function setWithExpiry(key, value, ttl) {
        if (ttl === 0) return
        const now = new Date()
        const expiryDay = ttl * 86400000
        const item = {
          value: value,
          expiry: now.getTime() + expiryDay,
        }
        localStorage.setItem(key, JSON.stringify(item))
      },

      get: function getWithExpiry(key) {
        const itemStr = localStorage.getItem(key)

        if (!itemStr) {
          return undefined
        }
        const item = JSON.parse(itemStr)
        const now = new Date()

        if (now.getTime() > item.expiry) {
          localStorage.removeItem(key)
          return undefined
        }
        return item.value
      }
    }
  
    win.getScript = url => new Promise((resolve, reject) => {
      const script = document.createElement('script')
      script.src = url
      script.async = true
      script.onerror = reject
      script.onload = script.onreadystatechange = function() {
        const loadState = this.readyState
        if (loadState && loadState !== 'loaded' && loadState !== 'complete') return
        script.onload = script.onreadystatechange = null
        resolve()
      }
      document.head.appendChild(script)
    })
  
      win.activateDarkMode = function () {
        document.documentElement.setAttribute('data-theme', 'dark')
        if (document.querySelector('meta[name="theme-color"]') !== null) {
          document.querySelector('meta[name="theme-color"]').setAttribute('content', '#0d0d0d')
        }
      }
      win.activateLightMode = function () {
        document.documentElement.setAttribute('data-theme', 'light')
        if (document.querySelector('meta[name="theme-color"]') !== null) {
          document.querySelector('meta[name="theme-color"]').setAttribute('content', '#ffffff')
        }
      }
      const t = saveToLocal.get('theme')
    
          if (t === 'dark') activateDarkMode()
          else if (t === 'light') activateLightMode()
        
      const asideStatus = saveToLocal.get('aside-status')
      if (asideStatus !== undefined) {
        if (asideStatus === 'hide') {
          document.documentElement.classList.add('hide-aside')
        } else {
          document.documentElement.classList.remove('hide-aside')
        }
      }
    
    const fontSizeVal = saveToLocal.get('global-font-size')
    if (fontSizeVal !== undefined) {
      document.documentElement.style.setProperty('--global-font-size', fontSizeVal + 'px')
    }
    
    const detectApple = () => {
      if (GLOBAL_CONFIG_SITE.isHome && /iPad|iPhone|iPod|Macintosh/.test(navigator.userAgent)){
        document.documentElement.classList.add('apple')
      }
    }
    detectApple()
    })(window)</script><meta name="generator" content="Hexo 5.4.0"><link rel="alternate" href="/atom.xml" title="全栈进阶那些事" type="application/atom+xml">
</head><body><div id="loading-box"><div class="loading-left-bg"></div><div class="loading-right-bg"></div><div class="spinner-box"><div class="configure-border-1"><div class="configure-core"></div></div><div class="configure-border-2"><div class="configure-core"></div></div><div class="loading-word">加载中...</div></div></div><div id="sidebar"><div id="menu-mask"></div><div id="sidebar-menus"><div class="avatar-img is-center"><img src="/img/zhiyao_avatar.jpg" onerror="onerror=null;src='/img/friend_404.gif'" alt="avatar"/></div><div class="site-data"><div class="data-item is-center"><div class="data-item-link"><a href="/archives/"><div class="headline">文章</div><div class="length-num">41</div></a></div></div><div class="data-item is-center"><div class="data-item-link"><a href="/tags/"><div class="headline">标签</div><div class="length-num">14</div></a></div></div><div class="data-item is-center"><div class="data-item-link"><a href="/categories/"><div class="headline">分类</div><div class="length-num">4</div></a></div></div></div><hr/><div class="menus_items"><div class="menus_item"><a class="site-page" href="/"><i class="fa-fw fas fa-home"></i><span> 首页</span></a></div><div class="menus_item"><a class="site-page" href="/archives/"><i class="fa-fw fas fa-archive"></i><span> 时间轴</span></a></div><div class="menus_item"><a class="site-page" href="/tags/"><i class="fa-fw fas fa-tags"></i><span> 标签</span></a></div><div class="menus_item"><a class="site-page" href="/categories/"><i class="fa-fw fas fa-folder-open"></i><span> 分类</span></a></div><div class="menus_item"><a class="site-page" href="/link/"><i class="fa-fw fas fa-link"></i><span> 友链</span></a></div><div class="menus_item"><a class="site-page" href="/about/"><i class="fa-fw fas fa-heart"></i><span> 关于</span></a></div></div></div></div><div class="post" id="body-wrap"><header class="not-top-img" id="page-header"><nav id="nav"><span id="blog_name"><a id="site-name" href="/">全栈进阶那些事</a></span><div id="menus"><div id="search-button"><a class="site-page social-icon search"><i class="fas fa-search fa-fw"></i><span> 搜索</span></a></div><div class="menus_items"><div class="menus_item"><a class="site-page" href="/"><i class="fa-fw fas fa-home"></i><span> 首页</span></a></div><div class="menus_item"><a class="site-page" href="/archives/"><i class="fa-fw fas fa-archive"></i><span> 时间轴</span></a></div><div class="menus_item"><a class="site-page" href="/tags/"><i class="fa-fw fas fa-tags"></i><span> 标签</span></a></div><div class="menus_item"><a class="site-page" href="/categories/"><i class="fa-fw fas fa-folder-open"></i><span> 分类</span></a></div><div class="menus_item"><a class="site-page" href="/link/"><i class="fa-fw fas fa-link"></i><span> 友链</span></a></div><div class="menus_item"><a class="site-page" href="/about/"><i class="fa-fw fas fa-heart"></i><span> 关于</span></a></div></div><div id="toggle-menu"><a class="site-page"><i class="fas fa-bars fa-fw"></i></a></div></div></nav></header><main class="layout" id="content-inner"><div id="post"><div id="post-info"><h1 class="post-title">Hadoop3.X分布式运行环境搭建手记</h1><div id="post-meta"><div class="meta-firstline"><span class="post-meta-date"><i class="fa-fw post-meta-icon far fa-calendar-alt"></i><span class="post-meta-label">发表于</span><time datetime="2020-11-07T05:33:37.000Z" title="undefined 2020-11-07 13:33:37">2020-11-07</time></span><span class="post-meta-categories"><span class="post-meta-separator">|</span><i class="fas fa-inbox fa-fw post-meta-icon"></i><a class="post-meta-categories" href="/categories/%E5%A4%A7%E6%95%B0%E6%8D%AE/">大数据</a></span></div><div class="meta-secondline"><span class="post-meta-separator">|</span><span class="post-meta-wordcount"><i class="far fa-file-word fa-fw post-meta-icon"></i><span class="post-meta-label">字数总计:</span><span class="word-count">2.5k</span><span class="post-meta-separator">|</span><i class="far fa-clock fa-fw post-meta-icon"></i><span class="post-meta-label">阅读时长:</span><span>11分钟</span></span><span class="post-meta-separator">|</span><span class="post-meta-pv-cv" id="" data-flag-title="Hadoop3.X分布式运行环境搭建手记"><i class="far fa-eye fa-fw post-meta-icon"></i><span class="post-meta-label">阅读量:</span><span id="busuanzi_value_page_pv"></span></span></div></div></div><article class="post-content" id="article-container"><h1 id="1-准备虚拟机（关闭防火墙、静态ip、主机名称）"><a href="#1-准备虚拟机（关闭防火墙、静态ip、主机名称）" class="headerlink" title="1.准备虚拟机（关闭防火墙、静态ip、主机名称）"></a>1.准备虚拟机（关闭防火墙、静态ip、主机名称）</h1><h2 id="1-1首先使用VMware安装CentOS7-4虚拟机"><a href="#1-1首先使用VMware安装CentOS7-4虚拟机" class="headerlink" title="1.1首先使用VMware安装CentOS7.4虚拟机"></a>1.1首先使用VMware安装CentOS7.4虚拟机</h2><p>安装好虚拟机后设置好静态IP(<code>192.168.126.120</code>,设置DNS)，<code>hostname</code>，执行<code>yum update</code>更新软件</p>
<p><img src="https://oscimg.oschina.net/oscnet/ff3261acae5d7d4926e17fa3ca3846dff55.jpg"></p>
<p><img src="https://oscimg.oschina.net/oscnet/e9fd7a37357136f3130d32fc08bd452846a.jpg"></p>
<p>这里由于进行了规划，所以提前设置了<code>hosts</code>，防止<code>ping hostname</code>的时候不同</p>
<p><img src="https://oscimg.oschina.net/oscnet/9c98ad2611f176c69bdef23c3bb74ab9498.jpg"></p>
<h2 id="1-2安装软件"><a href="#1-2安装软件" class="headerlink" title="1.2安装软件"></a>1.2安装软件</h2><p>安装<code>vim</code>，在<code>/usr/lib/jvm</code>目录安装好<code>jdk1.8.0_212</code>，并配置<code>/etc/profile</code>文件</p>
<p><img src="https://oscimg.oschina.net/oscnet/d68c2578976bb04775a02c15f733bdd35f6.jpg"></p>
<h2 id="1-3关闭防火墙"><a href="#1-3关闭防火墙" class="headerlink" title="1.3关闭防火墙"></a>1.3关闭防火墙</h2><h2 id="1-4创建hadoop用户"><a href="#1-4创建hadoop用户" class="headerlink" title="1.4创建hadoop用户"></a>1.4创建hadoop用户</h2><p>创建<code>hadoop</code>用户并配置<code>hadoop</code>用户具有<code>root</code>权限</p>
<p>在根目录”/“创建/data_disk/hadoop/tmp用于存储hadoop的数据</p>
<figure class="highlight makefile"><table><tr><td class="code"><pre><span class="line">mkdir -P /data/hadoop/tmp</span><br><span class="line">mkdir -P /data_disk/hadoop/data</span><br><span class="line">mkdir -P /data_disk/hadoop/logs</span><br><span class="line"></span><br></pre></td></tr></table></figure>

<h1 id="2-克隆虚拟机"><a href="#2-克隆虚拟机" class="headerlink" title="2.克隆虚拟机"></a>2.克隆虚拟机</h1><h2 id="2-1克隆两台虚拟机"><a href="#2-1克隆两台虚拟机" class="headerlink" title="2.1克隆两台虚拟机"></a>2.1克隆两台虚拟机</h2><h2 id="2-2更改克隆虚拟机的静态IP、hostname"><a href="#2-2更改克隆虚拟机的静态IP、hostname" class="headerlink" title="2.2更改克隆虚拟机的静态IP、hostname"></a>2.2更改克隆虚拟机的静态IP、hostname</h2><p>虚拟机2的IP：192.168.126.122，hostname：node102</p>
<p>虚拟机3的IP：192.168.126.123，hostname：node103</p>
<h1 id="3-编写集群分发脚本xsync"><a href="#3-编写集群分发脚本xsync" class="headerlink" title="3.编写集群分发脚本xsync"></a>3.编写集群分发脚本xsync</h1><p>在/home/hadoop目录下创建bin目录，并在bin目录下xsync创建文件</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">#!/bin/bash</span><br><span class="line">#1 获取输入参数个数，如果没有参数，直接退出</span><br><span class="line">pcount=$#</span><br><span class="line">if((pcount==0)); then</span><br><span class="line">echo no args;</span><br><span class="line">exit;</span><br><span class="line">fi</span><br><span class="line"></span><br><span class="line">#2 获取文件名称</span><br><span class="line">p1=$1</span><br><span class="line">fname=`basename $p1`</span><br><span class="line">echo fname=$fname</span><br><span class="line"></span><br><span class="line">#3 获取上级目录到绝对路径</span><br><span class="line">pdir=`cd -P $(dirname $p1); pwd`</span><br><span class="line">echo pdir=$pdir</span><br><span class="line"></span><br><span class="line">#4 获取当前用户名称</span><br><span class="line">user=`whoami`</span><br><span class="line"></span><br><span class="line">#5 循环</span><br><span class="line">for((host=101; host&lt;104; host++)); do</span><br><span class="line">        echo ------------------- hadoop-node$host --------------</span><br><span class="line">        rsync -rvl $pdir/$fname $user@node$host:$pdir</span><br><span class="line">done</span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>修改脚本<code>xsync</code>具有执行权限<code>chmod 777 xsync</code></p>
<p>注意：如果将<code>xsync</code>放到<code>/home/hadoop/bin</code>目录下仍然不能实现全局使用，可以将<code>xsync</code>移动到<code>/usr/local/bin</code>目录下。</p>
<h1 id="4-集群配置"><a href="#4-集群配置" class="headerlink" title="4.集群配置"></a>4.集群配置</h1><h2 id="1-集群规划"><a href="#1-集群规划" class="headerlink" title="1.集群规划"></a>1.集群规划</h2><p><img src="https://oscimg.oschina.net/oscnet/c646ab2c9196c3eacc49b81b10d47f7a191.jpg"></p>
<h2 id="2-安装hadoop"><a href="#2-安装hadoop" class="headerlink" title="2.安装hadoop"></a>2.安装hadoop</h2><p>在/opt/module下创建hadoop目录，使用ftp同步下载的hadoop3.1.4并解压，并且将<code>/opt/module/hadoop3.1.4</code>所属用户和所属组更改为<code>hadoop</code></p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">chown hadoop:hadoop -R /opt/module/hadoop3.1.4</span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>并配置<code>/etc/profile</code>文件，并且执行<code>source /etc/profile</code>立即生效</p>
<p><img src="https://oscimg.oschina.net/oscnet/ce418a94b46d0f3388b0e0e0287c936de3f.jpg"></p>
<h2 id="3-修改hadoop配置文件"><a href="#3-修改hadoop配置文件" class="headerlink" title="3.修改hadoop配置文件"></a>3.修改hadoop配置文件</h2><p>hadoop的配置文件都在/opt/module/hadoop3.1.4/etc/hadoop下面</p>
<p>（1）默认配置文件：</p>
<p><img src="https://oscimg.oschina.net/oscnet/up-5d1f35c43c4a078188d8f7e25901c4f35a2.png"></p>
<p>（2）自定义配置文件：</p>
<p><strong>core-site.xml<strong><strong>、hdfs-site.xml</strong></strong>、yarn-site.xml****、mapred-site.xml</strong>四个配置文件存放在$HADOOP_HOME/etc/hadoop这个路径上，用户可以根据项目需求重新进行修改配置。</p>
<h3 id="3-1核心配置文件core-site-xml"><a href="#3-1核心配置文件core-site-xml" class="headerlink" title="3.1核心配置文件core-site.xml"></a>3.1核心配置文件core-site.xml</h3><figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 hadoop]$ vim core-site.xml</span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>在该文件中编写如下配置</p>
<figure class="highlight xml"><table><tr><td class="code"><pre><span class="line"><span class="comment">&lt;!-- 指定NameNode的地址 --&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">property</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">name</span>&gt;</span>fs.defaultFS<span class="tag">&lt;/<span class="name">name</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">value</span>&gt;</span>hdfs://node101:8020<span class="tag">&lt;/<span class="name">value</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="name">property</span>&gt;</span></span><br><span class="line"></span><br><span class="line"><span class="comment">&lt;!-- 指定hadoop数据的存储目录 --&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">property</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">name</span>&gt;</span>hadoop.tmp.dir<span class="tag">&lt;/<span class="name">name</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">value</span>&gt;</span>/data_disk/hadoop/tmp<span class="tag">&lt;/<span class="name">value</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="name">property</span>&gt;</span></span><br><span class="line"></span><br><span class="line"><span class="comment">&lt;!-- 配置HDFS网页登录使用的静态用户为lytfly --&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">property</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">name</span>&gt;</span>hadoop.http.staticuser.user<span class="tag">&lt;/<span class="name">name</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">value</span>&gt;</span>lytfly<span class="tag">&lt;/<span class="name">value</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="name">property</span>&gt;</span></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<h3 id="3-2HDFS配置文件"><a href="#3-2HDFS配置文件" class="headerlink" title="3.2HDFS配置文件"></a>3.2HDFS配置文件</h3><p>配置hadoop-env.sh使用JDK1.8，不要使用JDK11，要不然启动Yarn和Hive等会因为JDK不兼容出现莫名其妙的错误。</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 hadoop]$ vim hadoop-env.sh</span><br><span class="line">export JAVA_HOME=/opt/module/jdk1.8.0.271</span><br></pre></td></tr></table></figure>

<p>因为jdk11默认没有activation.jar，所以需要手动下载jar放到${HADOOP_HOME}/share/hadoop/yarn/lib。</p>
<p>配置hdfs-site.xml</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 hadoop]$ vim hdfs-site.xml</span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>在该文件中编写如下配置，注意端口号！</p>
<figure class="highlight xml"><table><tr><td class="code"><pre><span class="line"><span class="meta">&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;</span></span><br><span class="line"><span class="meta">&lt;?xml-stylesheet type=&quot;text/xsl&quot; href=&quot;configuration.xsl&quot;?&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">configuration</span>&gt;</span></span><br><span class="line">    <span class="comment">&lt;!-- nn web端访问地址--&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">property</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">name</span>&gt;</span>dfs.namenode.http-address<span class="tag">&lt;/<span class="name">name</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">value</span>&gt;</span>node101:9870<span class="tag">&lt;/<span class="name">value</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;/<span class="name">property</span>&gt;</span></span><br><span class="line">    <span class="comment">&lt;!-- 2nn web端访问地址--&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">property</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">name</span>&gt;</span>dfs.namenode.secondary.http-address<span class="tag">&lt;/<span class="name">name</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">value</span>&gt;</span>node103:9868<span class="tag">&lt;/<span class="name">value</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;/<span class="name">property</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="name">configuration</span>&gt;</span></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<h3 id="3-3YARN配置文件"><a href="#3-3YARN配置文件" class="headerlink" title="3.3YARN配置文件"></a>3.3YARN配置文件</h3><p>配置yarn-site.xml</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 hadoop]$ vim yarn-site.xml</span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>在该文件中增加如下配置</p>
<figure class="highlight xml"><table><tr><td class="code"><pre><span class="line"><span class="meta">&lt;?xml version=&quot;1.0&quot;?&gt;</span></span><br><span class="line"><span class="meta">&lt;?xml-stylesheet type=&quot;text/xsl&quot; href=&quot;configuration.xsl&quot;?&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">configuration</span>&gt;</span></span><br><span class="line">    <span class="comment">&lt;!-- 指定MR走shuffle --&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">property</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">name</span>&gt;</span>yarn.nodemanager.aux-services<span class="tag">&lt;/<span class="name">name</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">value</span>&gt;</span>mapreduce_shuffle<span class="tag">&lt;/<span class="name">value</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;/<span class="name">property</span>&gt;</span></span><br><span class="line">    </span><br><span class="line">    <span class="comment">&lt;!-- 指定ResourceManager的地址--&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">property</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">name</span>&gt;</span>yarn.resourcemanager.hostname<span class="tag">&lt;/<span class="name">name</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">value</span>&gt;</span>node102<span class="tag">&lt;/<span class="name">value</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;/<span class="name">property</span>&gt;</span></span><br><span class="line">    </span><br><span class="line">    <span class="comment">&lt;!-- 环境变量的继承 --&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">property</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">name</span>&gt;</span>yarn.nodemanager.env-whitelist<span class="tag">&lt;/<span class="name">name</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">value</span>&gt;</span>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME<span class="tag">&lt;/<span class="name">value</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;/<span class="name">property</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="name">configuration</span>&gt;</span></span><br></pre></td></tr></table></figure>

<h3 id="3-4MapReduce配置文件"><a href="#3-4MapReduce配置文件" class="headerlink" title="3.4MapReduce配置文件"></a>3.4MapReduce配置文件</h3><p>配置mapred-site.xml</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 hadoop]$ cp mapred-site.xml.template mapred-site.xml</span><br><span class="line"></span><br><span class="line">[hadoop@node101 hadoop]$ vim mapred-site.xml</span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>在该文件中增加如下配置</p>
<figure class="highlight xml"><table><tr><td class="code"><pre><span class="line"><span class="meta">&lt;?xml version=&quot;1.0&quot;?&gt;</span></span><br><span class="line"><span class="meta">&lt;?xml-stylesheet type=&quot;text/xsl&quot; href=&quot;configuration.xsl&quot;?&gt;</span></span><br><span class="line"><span class="tag">&lt;<span class="name">configuration</span>&gt;</span></span><br><span class="line">    <span class="comment">&lt;!-- 指定MapReduce程序运行在Yarn上 --&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">property</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">name</span>&gt;</span>mapreduce.framework.name<span class="tag">&lt;/<span class="name">name</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">value</span>&gt;</span>yarn<span class="tag">&lt;/<span class="name">value</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;/<span class="name">property</span>&gt;</span></span><br><span class="line">    <span class="comment">&lt;!-- 历史服务器web端地址 --&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">property</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">name</span>&gt;</span>mapreduce.jobhistory.webapp.address<span class="tag">&lt;/<span class="name">name</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">value</span>&gt;</span>node101:19888<span class="tag">&lt;/<span class="name">value</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;/<span class="name">property</span>&gt;</span></span><br><span class="line">    <span class="comment">&lt;!-- 历史服务器端地址 --&gt;</span></span><br><span class="line">    <span class="tag">&lt;<span class="name">property</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">name</span>&gt;</span>mapreduce.jobhistory.address<span class="tag">&lt;/<span class="name">name</span>&gt;</span></span><br><span class="line">        <span class="tag">&lt;<span class="name">value</span>&gt;</span>node101:10020<span class="tag">&lt;/<span class="name">value</span>&gt;</span></span><br><span class="line">    <span class="tag">&lt;/<span class="name">property</span>&gt;</span></span><br><span class="line"><span class="tag">&lt;/<span class="name">configuration</span>&gt;</span></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>至此，hadoop集群所需要修改的配置文件都修改完成了</p>
<h2 id="4-在集群上分发配置好的hadoop配置文件"><a href="#4-在集群上分发配置好的hadoop配置文件" class="headerlink" title="4.在集群上分发配置好的hadoop配置文件"></a>4.在集群上分发配置好的hadoop配置文件</h2><figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 hadoop]$ xsync /opt/module/hadoop3.1.4/</span><br></pre></td></tr></table></figure>

<h1 id="5-集群单点启动"><a href="#5-集群单点启动" class="headerlink" title="5.集群单点启动"></a>5.集群单点启动</h1><p>这样启动方式还是比较麻烦的，在下一节会使用群起集群的功能</p>
<p>如果集群是第一次启动，需要<strong>格式化****NameNode</strong></p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 tmp]$ hdfs namenode -format</span><br></pre></td></tr></table></figure>

<p>在node101上启动NameNode</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 hadoop3.1.4]$ hadoop-daemon.sh start namenode</span><br><span class="line">[hadoop@node101 hadoop3.1.4]$ jps</span><br><span class="line">3461 NameNode</span><br></pre></td></tr></table></figure>

<p>在node101、node102以及node103上分别启动DataNode</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 tmp]$ hadoop-daemon.sh start datanode</span><br><span class="line">[hadoop@node101 tmp]$ jps</span><br><span class="line">3461 NameNode</span><br><span class="line">3608 Jps</span><br><span class="line">3561 DataNode</span><br><span class="line">[hadoop@node102 tmp]$ hadoop-daemon.sh start datanode</span><br><span class="line">[hadoop@node102 tmp]$ jps</span><br><span class="line">3190 DataNode</span><br><span class="line">3279 Jps</span><br><span class="line">[hadoop@node103 tmp]$ hadoop-daemon.sh start datanode</span><br><span class="line">[hadoop@node103 tmp]$ jps</span><br><span class="line">3237 Jps</span><br><span class="line">3163 DataNode</span><br></pre></td></tr></table></figure>

<p>启动报错：如果启动DataNode时报错：Initialization failed for Block pool (Datanode Uuid unassigned)</p>
<p>是因为namenode和datanode的clusterID不一致导致datanode无法启动。删除data、tmp、namenode 数据后，重新格式化即可。</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node103 hadoop]$rm -rf tmp</span><br><span class="line">[hadoop@node103 hadoop]$mkdir tmp </span><br></pre></td></tr></table></figure>

<h1 id="6-配置集群群起功能"><a href="#6-配置集群群起功能" class="headerlink" title="6.配置集群群起功能"></a>6.配置集群群起功能</h1><h2 id="6-1SSH无密登录配置"><a href="#6-1SSH无密登录配置" class="headerlink" title="6.1SSH无密登录配置"></a>6.1SSH无密登录配置</h2><p><img src="https://oscimg.oschina.net/oscnet/33d4e7e98c3328527b925017ac8fc22b6b9.jpg"></p>
<p>在node101生成公钥是密钥</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 hadoop]$ ssh-keygen -t rsa</span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>然后敲（三个回车），就会生成两个文件id_rsa（私钥）、id_rsa.pub（公钥）</p>
<p>将公钥拷贝到要免密登录的目标机器上</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 .ssh]$ ssh-copy-id node101</span><br><span class="line">[hadoop@node101 .ssh]$ ssh-copy-id node102</span><br><span class="line">[hadoop@node101 .ssh]$ ssh-copy-id node103</span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>同样的操作，需要在node102、node103机器上执行ssh免密设置。</p>
<p>.ssh文件夹下（~/.ssh）的文件功能解释</p>
<p><img src="https://oscimg.oschina.net/oscnet/bdbc1b95e271336e45b577e158413ced340.jpg"></p>
<h2 id="6-2群起集群配置"><a href="#6-2群起集群配置" class="headerlink" title="6.2群起集群配置"></a>6.2群起集群配置</h2><p>配置works，hadoop2.x版本是slaves，注意区分。</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 hadoop]$ vim /opt/module/hadoop3.1.4/etc/hadoop/works</span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>在该文件中增加如下内容：</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">node101</span><br><span class="line">node102</span><br><span class="line">node103</span><br></pre></td></tr></table></figure>

<p>注意：该文件中添加的内容结尾不允许有空格，文件中不允许有空行。</p>
<p>同步所有节点配置文件</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 hadoop]$ xsync works</span><br></pre></td></tr></table></figure>

<h2 id="6-3启动集群"><a href="#6-3启动集群" class="headerlink" title="6.3启动集群"></a>6.3启动集群</h2><p>如果集群是第一次启动，需要格式化NameNode（注意格式化之前，一定要先停止上次启动的所有namenode和datanode进程，然后再删除data和log数据）</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 hadoop3.1.4]$ bin/hdfs namenode -format</span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>启动HDFS</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 hadoop3.1.4]$ sbin/start-dfs.sh</span><br><span class="line">[hadoop@node101 hadoop3.1.4]$ jps</span><br><span class="line">4166 NameNode</span><br><span class="line">4482 Jps</span><br><span class="line">4263 DataNode</span><br><span class="line"></span><br><span class="line">[hadoop@node102 hadoop3.1.4]$ jps</span><br><span class="line">3218 DataNode</span><br><span class="line">3288 Jps</span><br><span class="line"></span><br><span class="line">[hadoop@node103 hadoop3.1.4]$ jps</span><br><span class="line">3221 DataNode</span><br><span class="line">3283 SecondaryNameNode</span><br><span class="line">3364 Jps</span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>启动YARN</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node102 hadoop3.1.4]$ sbin/start-yarn.sh</span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>注意：NameNode和ResourceManger如果不是同一台机器，不能在NameNode上启动 YARN，应该在ResouceManager所在的机器上启动YARN。</p>
<p>Web端查看NameNode<br>（a）浏览器中输入：<a target="_blank" rel="noopener external nofollow noreferrer" href="http://node101:9870/">http://node101:9870/</a></p>
<p><img src="https://oscimg.oschina.net/oscnet/up-8b77094f85c0e5cd3513bab709bba109016.png"></p>
<h1 id="7-集群测试"><a href="#7-集群测试" class="headerlink" title="7.集群测试"></a>7.集群测试</h1><h2 id="7-1上传文件到集群"><a href="#7-1上传文件到集群" class="headerlink" title="7.1上传文件到集群"></a>7.1上传文件到集群</h2><figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[hadoop@node101 data]$ hdfs dfs -mkdir -p /user/lyt/input</span><br><span class="line">[hadoop@node101 data]$ hdfs dfs -put wcinput/wc.input /user/lyt/input</span><br></pre></td></tr></table></figure>

<h1 id="8-集群启动-停止方式总结"><a href="#8-集群启动-停止方式总结" class="headerlink" title="8.集群启动/停止方式总结"></a>8.集群启动/停止方式总结</h1><h2 id="8-1各个服务组件逐一启动-停止"><a href="#8-1各个服务组件逐一启动-停止" class="headerlink" title="8.1各个服务组件逐一启动/停止"></a>8.1各个服务组件逐一启动/停止</h2><p>（1）分别启动/停止HDFS组件<br>        <a target="_blank" rel="noopener external nofollow noreferrer" href="http://hadoop-daemon.sh/">hadoop-daemon.sh</a>  start / stop  namenode / datanode / secondarynamenode<br>    （2）启动/停止YARN<br>        <a target="_blank" rel="noopener external nofollow noreferrer" href="http://yarn-daemon.sh/">yarn-daemon.sh</a>  start / stop  resourcemanager / nodemanager</p>
<h2 id="8-2各个模块分开启动-停止（配置ssh是前提）常用"><a href="#8-2各个模块分开启动-停止（配置ssh是前提）常用" class="headerlink" title="8.2各个模块分开启动/停止（配置ssh是前提）常用"></a>8.2各个模块分开启动/停止（配置ssh是前提）常用</h2><p>（1）整体启动/停止HDFS<br>        <a target="_blank" rel="noopener external nofollow noreferrer" href="http://start-dfs.sh/">start-dfs.sh</a>   /  <a target="_blank" rel="noopener external nofollow noreferrer" href="http://stop-dfs.sh/">stop-dfs.sh</a><br>    （2）整体启动/停止YARN<br>        <a target="_blank" rel="noopener external nofollow noreferrer" href="http://start-yarn.sh/">start-yarn.sh</a>  /  <a target="_blank" rel="noopener external nofollow noreferrer" href="http://stop-yarn.sh/">stop-yarn.sh</a></p>
<h2 id="8-3配置集群群起脚本"><a href="#8-3配置集群群起脚本" class="headerlink" title="8.3配置集群群起脚本"></a>8.3配置集群群起脚本</h2><p>如果不想单个启动的话，可以编写个shell脚本批量处理，参见<a target="_blank" rel="noopener external nofollow noreferrer" href="https://my.oschina.net/liuyuantao/blog/5268027">《Hadoop常用脚本》</a></p>
<h1 id="9-集群时间同步"><a href="#9-集群时间同步" class="headerlink" title="9.集群时间同步"></a>9.集群时间同步</h1><h2 id="9-1时间服务器配置（必须root用户）"><a href="#9-1时间服务器配置（必须root用户）" class="headerlink" title="9.1时间服务器配置（必须root用户）"></a>9.1时间服务器配置（必须root用户）</h2><p>（1）检查ntp是否安装</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[root@node101 hadoop]# rpm -qa|grep ntp</span><br><span class="line">ntp-4.2.6p5-10.el6.centos.x86_64</span><br><span class="line">fontpackages-filesystem-1.41-1.1.el6.noarch</span><br><span class="line">ntpdate-4.2.6p5-10.el6.centos.x86_64</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>如果没有安装，执行yum安装即可</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[root@node101 hadoop]#yum -y install ntp</span><br><span class="line">[root@node101 hadoop]#systemctl enable ntpd</span><br><span class="line">[root@node101 hadoop]#systemctl start ntpd</span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>（2）修改ntp配置文件</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[root@node101 hadoop]# vim /etc/ntp.conf</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>修改内容如下<br>a）修改1</p>
<p>取消注释，大约在17行（授权192.168.126.0-192.168.126.255网段上的所有机器可以从这台机器上查询和同步时间）</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">#restrict 192.168.126.0 mask 255.255.255.0 nomodify notrap为</span><br><span class="line">restrict 192.168.126.0 mask 255.255.255.0 nomodify notrap</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>b）修改2</p>
<p>注释掉配置，大约在21-24行（集群在局域网中，不使用其他互联网上的时间）</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">server 0.centos.pool.ntp.org iburst</span><br><span class="line">server 1.centos.pool.ntp.org iburst</span><br><span class="line">server 2.centos.pool.ntp.org iburst</span><br><span class="line">server 3.centos.pool.ntp.org iburst为</span><br><span class="line">#server 0.centos.pool.ntp.org iburst</span><br><span class="line">#server 1.centos.pool.ntp.org iburst</span><br><span class="line">#server 2.centos.pool.ntp.org iburst</span><br><span class="line">#server 3.centos.pool.ntp.org iburst</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>c）添加3</p>
<p>文件末尾添加（当该节点丢失网络连接，依然可以采用本地时间作为时间服务器为集群中的其他节点提供时间同步）</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">server 127.127.1.0</span><br><span class="line">fudge 127.127.1.0 stratum 10</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>（3）修改/etc/sysconfig/ntpd 文件</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[root@node101 hadoop]# vim /etc/sysconfig/ntpd</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>增加内容如下（让硬件时间与系统时间一起同步）</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">SYNC_HWCLOCK=yes</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>（4）重新启动ntpd服务</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[root@node101 hadoop]# service ntpd status</span><br><span class="line">ntpd 已停</span><br><span class="line">[root@node101 hadoop]# service ntpd start</span><br><span class="line">正在启动 ntpd：                                            [确定]</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>（5）设置ntpd服务开机启动</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[root@node101 hadoop]# chkconfig ntpd on</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<h2 id="9-2其他机器配置（必须root用户）"><a href="#9-2其他机器配置（必须root用户）" class="headerlink" title="9.2其他机器配置（必须root用户）"></a>9.2其他机器配置（必须root用户）</h2><p>（1）在其他机器配置10分钟与时间服务器同步一次</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[root@node102 hadoop]# crontab -e</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>编写定时任务如下：</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">*/10 * * * * /usr/sbin/ntpdate node101</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>（2）修改任意机器时间</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[root@node102 hadoop]# date -s &quot;2019-9-17 11:11:11&quot;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>（3）十分钟后查看机器是否与时间服务器同步</p>
<figure class="highlight plaintext"><table><tr><td class="code"><pre><span class="line">[root@node102 hadoop]# date</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>

<p>说明：测试的时候可以将10分钟调整为1分钟，节省时间。</p>
</article><div class="post-copyright"><div class="post-copyright__author"><span class="post-copyright-meta">文章作者: </span><span class="post-copyright-info"><a href="mailto:undefined" rel="external nofollow noreferrer">liuyuantao</a></span></div><div class="post-copyright__type"><span class="post-copyright-meta">文章链接: </span><span class="post-copyright-info"><a href="https://liuyuantao.gitee.io/2020/11/07/fd53f751c2eb.html">https://liuyuantao.gitee.io/2020/11/07/fd53f751c2eb.html</a></span></div><div class="post-copyright__notice"><span class="post-copyright-meta">版权声明: </span><span class="post-copyright-info">本博客所有文章除特别声明外，均采用 <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/" rel="external nofollow noreferrer" target="_blank">CC BY-NC-SA 4.0</a> 许可协议。转载请注明来自 <a href="https://liuyuantao.gitee.io" target="_blank">全栈进阶那些事</a>！</span></div></div><div class="tag_share"><div class="post-meta__tag-list"><a class="post-meta__tags" href="/tags/Hadoop/">Hadoop</a></div><div class="post_share"><div class="social-share" data-image="https://oscimg.oschina.net/oscnet/c646ab2c9196c3eacc49b81b10d47f7a191.jpg" data-sites="facebook,twitter,wechat,weibo,qq"></div><link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/social-share.js/dist/css/share.min.css" media="print" onload="this.media='all'"><script src="https://cdn.jsdelivr.net/npm/social-share.js/dist/js/social-share.min.js" defer></script></div></div><div class="post-reward"><div class="reward-button button--animated"><i class="fas fa-qrcode"></i> 打赏</div><div class="reward-main"><ul class="reward-all"><li class="reward-item"><a href="/img/qr_code_wechat.jpg" target="_blank"><img class="post-qr-code-img" src="/img/qr_code_wechat.jpg" alt="微信"/></a><div class="post-qr-code-desc">微信</div></li><li class="reward-item"><a href="/img/qr_code_alipay.jpg" target="_blank"><img class="post-qr-code-img" src="/img/qr_code_alipay.jpg" alt="支付宝"/></a><div class="post-qr-code-desc">支付宝</div></li></ul></div></div><nav class="pagination-post" id="pagination"><div class="prev-post pull-left"><a href="/2021/07/12/1e41bdf3e092.html"><img class="prev-cover" src="https://img-blog.csdnimg.cn/20200518143448459.png" onerror="onerror=null;src='/img/404.jpg'" alt="cover of previous post"><div class="pagination-info"><div class="label">上一篇</div><div class="prev_info">一文彻底搞懂Zookeeper</div></div></a></div><div class="next-post pull-right"><a href="/2020/11/07/a77fdbeecf9e.html"><img class="next-cover" src="https://oscimg.oschina.net/oscnet/8310dbed7d375aa31fd2cd1111458b5e854.png" onerror="onerror=null;src='/img/404.jpg'" alt="cover of next post"><div class="pagination-info"><div class="label">下一篇</div><div class="next_info">Hadoop分布式环境高可用配置</div></div></a></div></nav><div class="relatedPosts"><div class="headline"><i class="fas fa-thumbs-up fa-fw"></i><span>相关推荐</span></div><div class="relatedPosts-list"><div><a href="/2020/11/07/7d14a19c4606.html" title="DataNode工作机制"><img class="cover" src="https://oscimg.oschina.net/oscnet/up-0a831a2854d1859bc355a98a741ee74452b.png" alt="cover"><div class="content is-center"><div class="date"><i class="far fa-calendar-alt fa-fw"></i> 2020-11-07</div><div class="title">DataNode工作机制</div></div></a></div><div><a href="/2020/11/07/9cc4868ea813.html" title="Hadoop-Yarn常用的调优参数"><img class="cover" src="https://cdn.pixabay.com/photo/2021/08/08/10/34/ocean-6530523__340.jpg" alt="cover"><div class="content is-center"><div class="date"><i class="far fa-calendar-alt fa-fw"></i> 2020-11-07</div><div class="title">Hadoop-Yarn常用的调优参数</div></div></a></div><div><a href="/2020/11/07/5f9b1cc664c1.html" title="HDFS—集群扩容及缩容"><img class="cover" src="https://oscimg.oschina.net/oscnet/up-acb08d98e431429592cdb8802409bb29a19.png" alt="cover"><div class="content is-center"><div class="date"><i class="far fa-calendar-alt fa-fw"></i> 2020-11-07</div><div class="title">HDFS—集群扩容及缩容</div></div></a></div><div><a href="/2020/11/07/a8fda6be4c8d.html" title="HDFS—存储优化（纠删码）"><img class="cover" src="https://oscimg.oschina.net/oscnet/up-87ba27811580330155a29254c76223dd59f.png" alt="cover"><div class="content is-center"><div class="date"><i class="far fa-calendar-alt fa-fw"></i> 2020-11-07</div><div class="title">HDFS—存储优化（纠删码）</div></div></a></div><div><a href="/2020/11/07/3a181e232503.html" title="Hadoop小文件优化方法"><img class="cover" src="https://oscimg.oschina.net/oscnet/up-1681b15baa2d4bc66ceb813b9d6fcd4ad55.png" alt="cover"><div class="content is-center"><div class="date"><i class="far fa-calendar-alt fa-fw"></i> 2020-11-07</div><div class="title">Hadoop小文件优化方法</div></div></a></div><div><a href="/2020/11/07/a77fdbeecf9e.html" title="Hadoop分布式环境高可用配置"><img class="cover" src="https://oscimg.oschina.net/oscnet/8310dbed7d375aa31fd2cd1111458b5e854.png" alt="cover"><div class="content is-center"><div class="date"><i class="far fa-calendar-alt fa-fw"></i> 2020-11-07</div><div class="title">Hadoop分布式环境高可用配置</div></div></a></div></div></div></div><div class="aside-content" id="aside-content"><div class="sticky_layout"><div class="card-widget" id="card-toc"><div class="item-headline"><i class="fas fa-stream"></i><span>目录</span></div><div class="toc-content"><ol class="toc"><li class="toc-item toc-level-1"><a class="toc-link" href="#1-%E5%87%86%E5%A4%87%E8%99%9A%E6%8B%9F%E6%9C%BA%EF%BC%88%E5%85%B3%E9%97%AD%E9%98%B2%E7%81%AB%E5%A2%99%E3%80%81%E9%9D%99%E6%80%81ip%E3%80%81%E4%B8%BB%E6%9C%BA%E5%90%8D%E7%A7%B0%EF%BC%89"><span class="toc-text">1.准备虚拟机（关闭防火墙、静态ip、主机名称）</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#1-1%E9%A6%96%E5%85%88%E4%BD%BF%E7%94%A8VMware%E5%AE%89%E8%A3%85CentOS7-4%E8%99%9A%E6%8B%9F%E6%9C%BA"><span class="toc-text">1.1首先使用VMware安装CentOS7.4虚拟机</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#1-2%E5%AE%89%E8%A3%85%E8%BD%AF%E4%BB%B6"><span class="toc-text">1.2安装软件</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#1-3%E5%85%B3%E9%97%AD%E9%98%B2%E7%81%AB%E5%A2%99"><span class="toc-text">1.3关闭防火墙</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#1-4%E5%88%9B%E5%BB%BAhadoop%E7%94%A8%E6%88%B7"><span class="toc-text">1.4创建hadoop用户</span></a></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#2-%E5%85%8B%E9%9A%86%E8%99%9A%E6%8B%9F%E6%9C%BA"><span class="toc-text">2.克隆虚拟机</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#2-1%E5%85%8B%E9%9A%86%E4%B8%A4%E5%8F%B0%E8%99%9A%E6%8B%9F%E6%9C%BA"><span class="toc-text">2.1克隆两台虚拟机</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#2-2%E6%9B%B4%E6%94%B9%E5%85%8B%E9%9A%86%E8%99%9A%E6%8B%9F%E6%9C%BA%E7%9A%84%E9%9D%99%E6%80%81IP%E3%80%81hostname"><span class="toc-text">2.2更改克隆虚拟机的静态IP、hostname</span></a></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#3-%E7%BC%96%E5%86%99%E9%9B%86%E7%BE%A4%E5%88%86%E5%8F%91%E8%84%9A%E6%9C%ACxsync"><span class="toc-text">3.编写集群分发脚本xsync</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#4-%E9%9B%86%E7%BE%A4%E9%85%8D%E7%BD%AE"><span class="toc-text">4.集群配置</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#1-%E9%9B%86%E7%BE%A4%E8%A7%84%E5%88%92"><span class="toc-text">1.集群规划</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#2-%E5%AE%89%E8%A3%85hadoop"><span class="toc-text">2.安装hadoop</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#3-%E4%BF%AE%E6%94%B9hadoop%E9%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6"><span class="toc-text">3.修改hadoop配置文件</span></a><ol class="toc-child"><li class="toc-item toc-level-3"><a class="toc-link" href="#3-1%E6%A0%B8%E5%BF%83%E9%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6core-site-xml"><span class="toc-text">3.1核心配置文件core-site.xml</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#3-2HDFS%E9%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6"><span class="toc-text">3.2HDFS配置文件</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#3-3YARN%E9%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6"><span class="toc-text">3.3YARN配置文件</span></a></li><li class="toc-item toc-level-3"><a class="toc-link" href="#3-4MapReduce%E9%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6"><span class="toc-text">3.4MapReduce配置文件</span></a></li></ol></li><li class="toc-item toc-level-2"><a class="toc-link" href="#4-%E5%9C%A8%E9%9B%86%E7%BE%A4%E4%B8%8A%E5%88%86%E5%8F%91%E9%85%8D%E7%BD%AE%E5%A5%BD%E7%9A%84hadoop%E9%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6"><span class="toc-text">4.在集群上分发配置好的hadoop配置文件</span></a></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#5-%E9%9B%86%E7%BE%A4%E5%8D%95%E7%82%B9%E5%90%AF%E5%8A%A8"><span class="toc-text">5.集群单点启动</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#6-%E9%85%8D%E7%BD%AE%E9%9B%86%E7%BE%A4%E7%BE%A4%E8%B5%B7%E5%8A%9F%E8%83%BD"><span class="toc-text">6.配置集群群起功能</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#6-1SSH%E6%97%A0%E5%AF%86%E7%99%BB%E5%BD%95%E9%85%8D%E7%BD%AE"><span class="toc-text">6.1SSH无密登录配置</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#6-2%E7%BE%A4%E8%B5%B7%E9%9B%86%E7%BE%A4%E9%85%8D%E7%BD%AE"><span class="toc-text">6.2群起集群配置</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#6-3%E5%90%AF%E5%8A%A8%E9%9B%86%E7%BE%A4"><span class="toc-text">6.3启动集群</span></a></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#7-%E9%9B%86%E7%BE%A4%E6%B5%8B%E8%AF%95"><span class="toc-text">7.集群测试</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#7-1%E4%B8%8A%E4%BC%A0%E6%96%87%E4%BB%B6%E5%88%B0%E9%9B%86%E7%BE%A4"><span class="toc-text">7.1上传文件到集群</span></a></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#8-%E9%9B%86%E7%BE%A4%E5%90%AF%E5%8A%A8-%E5%81%9C%E6%AD%A2%E6%96%B9%E5%BC%8F%E6%80%BB%E7%BB%93"><span class="toc-text">8.集群启动&#x2F;停止方式总结</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#8-1%E5%90%84%E4%B8%AA%E6%9C%8D%E5%8A%A1%E7%BB%84%E4%BB%B6%E9%80%90%E4%B8%80%E5%90%AF%E5%8A%A8-%E5%81%9C%E6%AD%A2"><span class="toc-text">8.1各个服务组件逐一启动&#x2F;停止</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#8-2%E5%90%84%E4%B8%AA%E6%A8%A1%E5%9D%97%E5%88%86%E5%BC%80%E5%90%AF%E5%8A%A8-%E5%81%9C%E6%AD%A2%EF%BC%88%E9%85%8D%E7%BD%AEssh%E6%98%AF%E5%89%8D%E6%8F%90%EF%BC%89%E5%B8%B8%E7%94%A8"><span class="toc-text">8.2各个模块分开启动&#x2F;停止（配置ssh是前提）常用</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#8-3%E9%85%8D%E7%BD%AE%E9%9B%86%E7%BE%A4%E7%BE%A4%E8%B5%B7%E8%84%9A%E6%9C%AC"><span class="toc-text">8.3配置集群群起脚本</span></a></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#9-%E9%9B%86%E7%BE%A4%E6%97%B6%E9%97%B4%E5%90%8C%E6%AD%A5"><span class="toc-text">9.集群时间同步</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#9-1%E6%97%B6%E9%97%B4%E6%9C%8D%E5%8A%A1%E5%99%A8%E9%85%8D%E7%BD%AE%EF%BC%88%E5%BF%85%E9%A1%BBroot%E7%94%A8%E6%88%B7%EF%BC%89"><span class="toc-text">9.1时间服务器配置（必须root用户）</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#9-2%E5%85%B6%E4%BB%96%E6%9C%BA%E5%99%A8%E9%85%8D%E7%BD%AE%EF%BC%88%E5%BF%85%E9%A1%BBroot%E7%94%A8%E6%88%B7%EF%BC%89"><span class="toc-text">9.2其他机器配置（必须root用户）</span></a></li></ol></li></ol></div></div></div></div></main><footer id="footer"><div id="footer-wrap"><div class="copyright">&copy;2020 - 2021 By liuyuantao</div><div class="framework-info"><span>框架 </span><a target="_blank" rel="noopener external nofollow noreferrer" href="https://hexo.io">Hexo</a><span class="footer-separator">|</span><span>主题 </span><a target="_blank" rel="noopener external nofollow noreferrer" href="https://github.com/jerryc127/hexo-theme-butterfly">Butterfly</a></div><div class="footer_custom_text">扫描二维码,关注我们!了解更多全栈技能!<br/><img style="width:100px;height:100px;" src="/img/qrcode_for_gh_da91cebd00cc_258.jpg" ></div></div></footer></div><div id="rightside"><div id="rightside-config-hide"><button id="readmode" type="button" title="阅读模式"><i class="fas fa-book-open"></i></button><button id="font-plus" type="button" title="放大字体"><i class="fas fa-plus"></i></button><button id="font-minus" type="button" title="缩小字体"><i class="fas fa-minus"></i></button><button id="translateLink" type="button" title="简繁转换">简</button><button id="darkmode" type="button" title="浅色和深色模式转换"><i class="fas fa-adjust"></i></button><button id="hide-aside-btn" type="button" title="单栏和双栏切换"><i class="fas fa-arrows-alt-h"></i></button></div><div id="rightside-config-show"><button id="rightside_config" type="button" title="设置"><i class="fas fa-cog fa-spin"></i></button><button class="close" id="mobile-toc-button" type="button" title="目录"><i class="fas fa-list-ul"></i></button><button id="go-up" type="button" title="回到顶部"><i class="fas fa-arrow-up"></i></button></div></div><div id="local-search"><div class="search-dialog"><div class="search-dialog__title" id="local-search-title">本地搜索</div><div id="local-input-panel"><div id="local-search-input"><div class="local-search-box"><input class="local-search-box--input" placeholder="搜索文章" type="text"/></div></div></div><hr/><div id="local-search-results"></div><span class="search-close-button"><i class="fas fa-times"></i></span></div><div id="search-mask"></div></div><div><script src="/js/utils.js"></script><script src="/js/main.js"></script><script src="/js/tw_cn.js"></script><script src="https://cdn.jsdelivr.net/npm/medium-zoom/dist/medium-zoom.min.js"></script><script src="https://cdn.jsdelivr.net/npm/instant.page/instantpage.min.js" type="module"></script><script src="https://cdn.jsdelivr.net/npm/node-snackbar/dist/snackbar.min.js"></script><script src="/js/search/local-search.js"></script><script>var preloader = {
  endLoading: () => {
    document.body.style.overflow = 'auto';
    document.getElementById('loading-box').classList.add("loaded")
  },
  initLoading: () => {
    document.body.style.overflow = '';
    document.getElementById('loading-box').classList.remove("loaded")

  }
}
window.addEventListener('load',preloader.endLoading())</script><div class="js-pjax"></div><script defer="defer" id="fluttering_ribbon" mobile="false" src="https://cdn.jsdelivr.net/npm/butterfly-extsrc@1/dist/canvas-fluttering-ribbon.min.js"></script><script async data-pjax src="//busuanzi.ibruce.info/busuanzi/2.3/busuanzi.pure.mini.js"></script></div></body></html>